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Abstract 

The purposes of the research were (1) to develop the physical therapy practical assessment system by using the 
multisource feedback (MSF) approach and (2) to investigate the effectiveness of the implementation of the 
developed physical therapy practical assessment system. The development of physical therapy practical 
assessment system by using MSF was determined by nine experts in physical therapy. Suitability and feasibility 
of the system for each sub-group were investigated. Five input factors, two process factors, two output factors, 
and two feedback factors were involved in the system. Level of suitability and feasibility of elements in each 
sub-group presented at high to the highest level. In system testing, 40 physical therapy students were participated. 
Raters consisted of clinical educators, students (self-assessment), friends (students who in the same practical 
group), and patients. Twice assessments during the period of clinical practice were performed. Data analysis for 
generalizability coefficient (G-coefficient) was evaluated by the EduG program. Quality of the system was 
evaluated 4 aspects including the utility, feasibility, propriety, and accuracy. These were calculated by mean (x) 
and standard deviation (SD). The values of G-coefficient for absolute and relative were 0.86 and 0.88, 
respectively. In addition, quality of the system showed value at high to the highest level in all aspects. 
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1. Introduction 

Teaching and learning in higher education, there are many processes to facilitating knowledge, skills, and 
behaviors. Clinical practice is one of the most important processes to develop the students to be professional. 
This is the highest level of competency that students must achieve prior to graduation. Students are required to 
integrate and apply their knowledge and practical skill in classroom to real-life situations. This is very 
challenging for clinical educators to arrange the suitable situation and adapt or develop methods to motivate 
students to achieve the professional competence. In addition, the clinical practice management should consider 
several elements enriched in encourage student’s experience effectively, such as being a good role model of the 
clinical educator, teaching preparation, facilitate reflective learning, adequate learning resources, assessment 
system that consistent with the objective, and giving and receiving good feedback. Especially, feedback in 
clinical education should be specific which had provided information for narrowing the gap between actual and 
expected performances. The purpose of all is to promote students to success in clinical practice (Archer, 2010; 
Cunningham, Baird, & Wright, 2015; Emstzen, Bitzer, & Grimmer-Somers, 2009; Janicik & Fletcher, 2003; 
Jette, Nelson, Palalma, & Wetherbee, 2014; Ramani & Leinster, 2008). Physical therapy is one of the health 
professional team that provides health care for all people. Teaching in clinical practice are different in classroom 
learning because it occurs in real situation. Students are divided into small group of 3-5 persons and entrusted to 
the care of patients under clinical educator responsibility. 

However, not everyone can achieve the requirement by clinical practice. So, it is necessary to give students an 
opportunity to improve and develop their practical skill. In order to get the appropriate information for 
improving of student’s performance, the appropriate assessment method is required. Moreover, quality of clinical 
practical assessment is the most importance part which has to be concerned. Because clinical practical 
assessment is a key to succeed in health professions in the future, the formative assessment is benefit to help 
students to get valuable feedback and use this information in planning and developing their own practice in time. 
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Moreover, formative assessment can identify students learning to result in academic achievement in the desired 
direction (Andrade & Cizek, 2010; Black & Wiliam, 2009; Kahl, Hofman, & Bryant, 2013). In addition, these 
can be sharing attitude of teachers and students during teaching. But the most recent assessment in clinical 
practice is often used to judge performance of student’s practice (summative assessment). Formative assessment 
was only used in term of verbal or informal that can lead to misunderstand between clinical educators and 
students. These issues can be found in studies that reported perception of clinical educator providing feedback. 
The research suggested that the understanding of clinical educators were always providing feedback to their 
student frequently. On the other hand, students reported that the receiving feedback from clinical educators was 
rare. It might be due to an assessment was used that not enough to students perceive and understand (Archer, 
2010; Liberman, Liberman, Steinert, McLeod, & Meterissian, 2005; Van de Ridder, Stokking, McGaghie, & 
Cate, 2008). 

Multisource feedback (MSF) is one of the assessment tools for collection of detailed information from multiple 
sources which called the assessors. The MSF can be used to improve performance of an examinee in the clinical 
practice of medical education. The MSF can be applied in two manner including formative and summative 
assessments, but mostly used in the form of formative assessment. The application of MSF was usually used to 
assessment and report informal. The process of MSF will get valuable information of feedback which impacted 
to student’s performance directly. Therefore, there were several studies suggested that MSF was a tool for 
improving student’s competences such as communication, teamwork, patient management, and professional 
development (Bracken, Timmreck, & Church, 2001; Davies, Ciantar, Jubraj, & Bates, 2013; Violato, Worsfold, 
& Polgar, 2009). There are many elements involved when using MSF such as design and select instrument, 
select raters, collect data, analyze data, report feedback etc. These assessment can help students to know their 
strength and weak performances, similarly, the teachers can use the result to plan the teaching and learning for 
students. 

Furthermore, MSF will be received the result of assessment from the assessors more than one person. The 
assessors will consist of individuals who have relationships with the student’s practice e.g. clinical educator, peer, 
and patient. This is in line with the authentic situation that there are many people involving student’s practical 
experience. The different views of assessors will provide important information on the development of student’s 
performance. For examples, clinical educator views can identify strength and weak performance. Peers views 
who practice in same group that can explore behavior practice routine. In addition, patient views can give 
another valuable feedback because patients are an importance stakeholders in the future. This assessment will 
assist the students to ready for work and also serve as reflection of the curriculum quality. Moreover, MSF will 
promote the student to be one of the assessors (self-assessment). This is very useful for the students to assess 
themselves more precise and encourage student lifelong learning (Cox & Irby, 2007; Davies & Archer, 2005; 
Davies, Ciantar, Jubraj, & Bates, 2013; Overeem et al., 2010; Reinders et al., 2011; Wall, Singh, Whitehouse, 
Hassell, & Howes, 2012). However, there was no study about the application of MSF in formative manner for 
improving student performance in the clinical practice for physical therapy education. Therefore, as the first step, 
researchers interested in developing the physical therapy practical assessment system by using MSF and 
investigate the effectiveness of the developed system in order to improve quality of clinical education. The 
purpose of the research was (1) to develop physical therapy practical assessment by using MSF and (2) to 
investigate the quality of the implementation of the developed physical therapy practical assessment system. 

2. Method 

There were 2 steps of developing the system as described in the following; 

Step 1: Developing of the physical therapy practical assessment system by using MSF; the assessment system 
was developed by a synthesis of documents related to the assessment in clinical practice and MSF approach. 
Then, investigation of suitability and feasibility of the developing system were performed by nine experts. 

Step 2: Experiment of the developed physical therapy practical assessment system; after completed developing, 
the quality of the assessment system was evaluated. The participants in this step were mentioned as examinee. 
There were consisted of 40 physical therapy students of the Faculty of Physical Therapy, Mahidol University. 
The raters consisted of clinical educators, students (self-assessment), peers (students who in the same practical 
group), and patients. The instrument used for assessing students practice was a checklist format questionnaire. 
The topics of assessment were competence requirements in physical therapy profession, such as professional 
behavior, communication, and patient management. Validity and reliability of the questionnaire were tested and 
found to be high (0.87 for validity and 0.91 for reliability). 

Data analysis for generalization was using EduG program. In addition, quality of the system was evaluated in 4 
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aspects following standards of utility, feasibility, propriety, and accuracy. These were reported by mean (x) and 
standard deviation (SD). 

3. Results 

The results of development of physical therapy practical assessment system by using MSF were divided into 
subgroups including the input, process, output, and feedback. There were five input elements, two process 
elements, two output elements, and two feedback elements. The evaluation of subgroups by experts showed 
every element was at the highest to high levels. The five input elements were (1) objectives of assessment, (2) 
examinee and raters, (3) indicator, (4) tool, and (5) duration of assessment. The two process elements were (1) 
preparation, and (2) evaluation. The two output elements were (1) student’s strength and weak performance, and 
(2) suggestion for improve performance. The two feedback elements were (1) reporting outcome and feedback 
on practice to student, and (2) reporting outcome and feedback on practice to department of education. The 
suitability and feasibility evaluation indicated the values in high to the highest levels. The developed physical 
therapy practical assessment system by using MSF is shown in Figure 1. 



Figure 1. Process of physical therapy practical assessment by using multisource feedback 


The trial of physical therapy practical assessment system by using MSF was assessed two times in clinical 
practice period. Raters consisted of clinical educators, students, friends, and patients. The results of G-coefficient 
for absolute and relative by using EduG program were 0.86 and 0.88, respectively. Addition, the result of the 
decision of the study (D-study) for generalization in occasions between 1 and 5 showed in Table 1. 


Table 1. G-coefficient in different occasions of D-study 



n 0 (frequency of occasions) 




G-coefficient 

1 

2 

3 

4 

5 

2 

P Abs 

0.7566 

0.8614 

0.9031 

0.9256 

0.9491 

9 2 Rel 

0.7919 

0.8839 

0.9195 

0.9384 

0.9501 


Note. Bold numbers represent result from the experiment, regular number represent results from EduG program. 


Table 1 presented G-coefficient in different of occasions. The generalization for absolute (p 2 Abs) was 0.7566, 
0.8614, 0.9031, 0.9256, and 0.9491, respectively. The generalization for relative (p 2 R e i) was 0.7919, 0.8839, 
0.9195, 0.9384, and 0.9501, respectively. 

The quality of the developed physical therapy practical assessment system by using MSF was considered in 
system effectiveness. The standard of effectiveness of this system was evaluated in four aspects including the 
standards of utility, feasibility, propriety, and accuracy. The average and standard deviation values (x ± SD) of 
benefits, feasibility, suitable and accuracy of physical therapy practical assessment system by using MSF were 
4.40 ± 0.66, 4.30 ± 0.66, 4.36 ± 0.63 and 4.55 ± 0.53, respectively. 

4. Discussion 

According to the first step of the study, there were eleven elements and four groups of factor in the developed 
physical therapy practical assessment system. The four groups include input, process, output, and feedback as the 
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recommendations of research by Scott (2008). Group of elements had inter-related to function to making goals 
success. This study found that the system consisted of five input elements, two process elements, two output 
elements, and two feedback elements. The system suitability and feasibility evaluation by experts indicated that 
each sub-group had values in high to the highest levels. These indicated that the essential elements had been 
prepared when using this system. The input and process groups of system were support by studies as follows. 
The study of Mizikaci (2006) reported that the essential elements for quality in higher education were clarify of 
the aims and objectives of assessment. Research by Donabedian (1966) cited in Jette, Nelson, Palalma, and 
Wetherbee (2014) indicated that duration of experiences, evaluation of student’s performance, and model of 
supervisory were considered in quality of clinical education. Besides, the study by Caiman, Watson, Norman, 
Redfem, and Murrells (2002) presented that the assessing practice should be prepared of assessors and 
assessment method. In addition, student academic achievement always put in the output of evaluation in 
educational system. As the study by Jette, Nelson, Palalma, and Wetherbee (2014), which indicated the essential 
outcomes, should be concerned in student’s competencies and stakeholder satisfaction. It supports that the output 
of the system has appropriate. 

The results of G-coefficient for absolute and relative were 0.86 and 0.88, respectively. As the description by 
Shavelson and Webb (1991) explained that G-coefficient was used to interpret of reliability coefficient across 
various facets of the study. The results of the study indicated this system had reliability in high level. This was 
supported by Cunningham, Baird, and Wright (2015) that mentioned the assessment in clinical education has to 
adequate reliability. 

In addition, G-coefficient for absolute and relative in D-study which provided information for make decision in 
the most stable and efficient measurement procedures (Shavelson & Webb, 1991). The results of D-study 
presented that generalization for absolute (p 2 Abs) were 0.7566, 0.8614, 0.9031, 0.9256, and 0.9491, respectively. 
The generalization for relative (p 2 R e i) was 0.7919, 0.8839, 0.9195, 0.9384, and 0.9501, respectively. So, the 
appropriate implementation of this system should be used in assessment about two or three times because small 
different in G-coefficient was found since the third assessment. So, more than 3 times of assessment is not 
necessary. This is supported by Jette, Nelson, Palalma, and Wetherbee (2014), which presented the frequent of 
assessment, should be at least two times which usually performed at mid-point and final week of clinical practice 
periods. Excessive assessments may be waste in time and resource and resulted in unexpected outcomes. These 
could be found in previous studies that mentioned about negative impacts to the students. For example, it can 
make stress and uncomfortable that lead to disturb student’s learning in clinical experiences (Changiz, 
Malekpour, & Zargham-Boroujeni, 2012; Razaee, Esmaeili, & Habibzadeh, 2015). 

According to the joint committee on standards for educational evaluation (1994) recommended that the system 
effectiveness should be evaluated standards of utility, feasibility, propriety, and accuracy. This study found that 
the effectiveness of the developing of physical therapy practical assessment system by using MSF were at high 
to the highest levels of all aspects. Therefore, this system had adequate quality for implementing future physical 
therapy practical assessment. 

This study was included participants only in the Faculty of Physical Therapy, Mahidol University. Because there 
were differences in pattern of clinical practice and evaluation in each physical therapy institutes in Thailand, 
future study will be challenge to investigate this system used in other institutes or other health professional 
educations that required clinical practice. 
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